Processing Symbolic Sequences by Recurrent Neural Networks Trained by Kalman Filter-Based Algorithms

Kalman filter (KF)-based techniques used for recurrent neural networks (RNNs) training on real-valued time series have already shown their potential. On the other hand gradient descent approaches such as back-propagation through time (BPTT) or real-time recurrent learning (RTRL) algorithms are still widely used by researchers working with symbolic sequences. The aim of this work is to show how KF-based techniques used for training RNNs can deal with symbolic time series. Experiments with different data sets are described, comparing the next symbol prediction performance of RNNs trained with KF-based algorithms and classical gradient descent techniques. KF-based training approaches usually show better performance, faster convergence and higher robustness in comparing with standard algorithms.